Automated control of image acquisition via use of mobile device user interface

ABSTRACT

Techniques are described for using computing devices to perform automated operations to control acquisition of images in a defined area, including obtaining and using data from one or more hardware sensors on a mobile device that is acquiring the images, analyzing the sensor data (e.g., in a real-time manner) to determine the geometric orientation of the mobile device in three-dimensional (3D) space, and using that determined orientation to control the acquisition of further images by the mobile device. In some situations, the determined orientation information may be used in part to automatically generate and display a corresponding GUI (graphical user interface) that is overlaid on and augments displayed images of the environment surrounding the mobile device during the image acquisition process, so as to control the mobile device&#39;s geometric orientation in 3D space.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 16/920,770, filed Jul. 6, 2020 and entitled “Automated ControlOf Image Acquisition Via Use Of Mobile Device Interface”, which ishereby incorporated by reference in its entirety. U.S. patentapplication Ser. No. 16/920,770 is a continuation of U.S. patentapplication Ser. No. 16/236,187, filed Dec. 28, 2018 and entitled“Automated Control Of Image Acquisition Via Use Of Acquisition DeviceSensors”, now U.S. Pat. No. 10,708,507, which is hereby incorporated byreference in its entirety. U.S. patent application Ser. No. 16/236,187claims the benefit of U.S. Provisional Patent Application No.62/744,480, filed Oct. 11, 2018 and entitled “Automated Control Of ImageAcquisition Via Use Of Acquisition Device Sensors,” which is herebyincorporated by reference in its entirety

TECHNICAL FIELD

The following disclosure relates generally to techniques forautomatically controlling acquisition of images in a defined area foruse in generating mapping information for the area, as well as forsubsequently using the generated mapping information in one or moremanners, such as by real-time analysis and use of sensor data on amobile device that acquires the images.

BACKGROUND

In various fields and circumstances, such as architectural analysis,property inspection, real estate acquisition and development, generalcontracting, improvement cost estimation and other circumstances, it maybe desirable to view the interior of a house, office, or other buildingwithout having to physically travel to and enter the building. However,it can be difficult or impossible to effectively capture visualinformation within building interiors without using specializedequipment, as well as to display visual information captured withinbuilding interiors to users at remote locations, such as to enable auser to fully understand the layout and other details of the interiorand to control the display in a user-selected manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1D are diagrams depicting an exemplary building interiorenvironment and computing system(s) for use in embodiments of thepresent disclosure, including to generate and present informationrepresenting the building interior.

FIGS. 2A-2I illustrate examples of automatically analyzing and usingsensor data on a mobile device that is acquiring images from a buildinginterior in order to control the image acquisition.

FIG. 3 is a block diagram illustrating a computing system suitable forexecuting an embodiment of a system that performs at least some of thetechniques described in the present disclosure.

FIGS. 4A-4D illustrate an example embodiment of a flow diagram for anImage Capture and Analysis (ICA) system routine in accordance with anembodiment of the present disclosure.

FIGS. 5A-5B illustrate an example embodiment of a flow diagram for aMapping Information Generation Manager (MIGM) system routine inaccordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure describes techniques for using computing devicesto perform automated operations to control acquisition of images in adefined area, as well as techniques for further automated operations tosubsequently generate and use mapping information from the acquiredimages in one or more further manners. In at least some embodiments, theautomated control of the image acquisition includes obtaining and usingdata from one or more hardware sensors on a handheld mobile device thatis acquiring the images, analyzing the sensor data (e.g., in a real-timemanner) to determine the geometric orientation of the mobile device inthree-dimensional (3D) space (e.g., with respect to 3 degrees ofrotational freedom, referred to as pitch, roll and yaw, or swivel, tiltand pivot), and using that determined orientation to control theacquisition of further images by the mobile device. In some suchembodiments, the handheld mobile device may be carried by a user tomultiple viewing locations within the defined area (e.g., within abuilding or other structure) and used to acquire a panorama image (orother information) at each such viewing location, by the user turninghis or her body in a full or partial circle at that viewing locationwhile the images are acquired, and with the determined orientationinformation corresponding to an outward direction of the mobile device'scamera or other imaging sensors and being used in part to automaticallygenerate and display a corresponding GUI (graphical user interface) thatis overlaid on and augments displayed images of the environmentsurrounding the mobile device during the image acquisition process, soas to control the mobile device's geometric orientation in 3D space byguiding the placement of the mobile device for acquisition (orre-acquisition) of images at defined orientations from the viewinglocation that satisfy one or more defined criteria. Additional detailsare included below regarding the automated control of the imageacquisition using sensor data of the acquiring mobile device, and someor all of the corresponding techniques described herein may, in at leastsome embodiments, be performed via automated operations of an InteriorCapture and Analysis (“ICA”) system, as discussed further below.

As noted above, the images acquired from a multi-room building (or otherstructure) may be used in various manners in various embodiments,optionally along with additional metadata information obtained duringthe image acquisition, including to further generate and use mappinginformation for the building or other structure. For example, if thedefined area is an interior of a multi-room building (e.g., a house,office, etc.), the generated information may include a floor map of thebuilding, and/or multiple inter-connected panorama images or otherimages acquired at various viewing locations within the building. Such agenerated floor map and/or other generated mapping-related informationmay be further used in various manners in various embodiments, includingfor controlling navigation of other devices (e.g., autonomous vehicles),for display on one or more client devices in corresponding GUIs(graphical user interfaces), etc. Additional details are included belowregarding the automated generation and use of mapping information, andsome or all of the corresponding techniques described herein may, in atleast some embodiments, be performed via automated operations of aMapping Information Generation Manager (“MIGM”) system, as discussedfurther below.

In at least some embodiments and situations, some or all of the imagesacquired for a building (or other structure) may be panorama images thatare each acquired at one of multiple viewing locations in or around thebuilding, such as to optionally generate a panorama image at a viewinglocation from a video at that viewing location (e.g., a 360° video takenfrom a smartphone or other mobile device held by a user turning at thatviewing location), from multiple images acquired in multipledirections/orientations from the viewing location (e.g., from asmartphone or other mobile device held by a user turning at that viewinglocation), etc. It will be appreciated that such a panorama image may insome situations be represented in a spherical coordinate system andcover up to 360° around horizontal and/or vertical axes, such that anend user viewing a panorama image may move the viewing direction withinthe panorama image to different directions to cause different images (or“views”) to be rendered within the panorama image (including, if thepanorama image is represented in a spherical coordinate system, toconvert the image being rendered into a planar coordinate system).Furthermore, acquisition metadata regarding the capture of such panoramaimages may be obtained and used in various manners, such as dataacquired from IMU (inertial measurement unit) sensors (e.g., one or moreaccelerometers, gyroscopes, compasses, etc.) or other sensors (e.g.,imaging sensors) of a mobile device as it is carried by a user orotherwise moved between viewing locations.

As noted above, in at least some embodiments, the automated control ofthe image acquisition includes displaying a GUI that is overlaid onimages acquired from a camera or other imaging system of a handheldmobile device as a user rotates around a vertical axis at a viewinglocation, with the GUI including visual indicators based on the mobiledevice's current position in 3D space (geometric orientation withrespect to tilt, pivot and swivel; and optionally location translationalong a vertical axis and perpendicular horizontal axes, such as withrespect a viewing location or other defined location in 3D space; andalso referred to herein more generally as “orientation” and optionally“location”)—as used herein, the term “position” corresponds to such 3Dgeometric orientation and optionally location, unless otherwiseindicated. In at least some embodiments, the display and updating of theGUI is performed in a continuous or substantially continuous manner asthe user turns, to provide real-time or near-real-time feedback based ondata being acquired from the mobile device's IMU hardware sensors and/orother hardware sensors (e.g., with the displayed GUI visually showingthe mobile device's current position within a period of time from thehardware sensor data availability that is measured in at least one ofmicroseconds, milliseconds, hundredths of a second, tenths of a second,a second, etc.). Thus, as a display on the mobile device (or otherdevice available to the user) shows the images available from the mobiledevice's imaging sensor(s), whether the images are being automaticallysaved or not (e.g., as continuous video), the display overlaid on theimages includes information that is sufficiently current to allow theuser to modify the position of the mobile device if the current positiondiffers from the target positions in 3D space (target geometricorientations, and optionally target locations) at which to acquire andsave images by an amount that exceeds a defined threshold. Additionaldetails are included below related to determining a mobile device'scurrent position in 3D space using data from hardware sensors on themobile device.

The form of the displayed GUI, and its contents, may include variousdisplayed icons or other elements in various embodiments, includinguser-selectable controls and instructional text or other visualindicators. In at least some embodiments, the displayed elements of theGUI include one or more visual indicators of the mobile device's currentposition, such as in the center of the underlying displayed image insome embodiments—examples of such visual current position indicators mayinclude an icon representing a smartphone or other mobile device, acrosshairs or other combination of at least one horizontal and/orvertical line (e.g., in a manner similar to an attitude indicator in anairplane's flight instruments), multiple horizontal lines (e.g., aboveand below the current position), multiple vertical lines (e.g., to theright and left of the current position), etc. In addition, in at leastsome embodiments, target positions for some or all acquired and savedimages may further be shown in the displayed GUI (e.g., for all suchtarget positions visible in the current image), such as to providefeedback regarding previous acquisitions that have occurred—in someembodiments, such visual indicators of previous target positions mayfurther include visual indications of the success or failure of theimage acquisition (e.g., one or more degrees of quality), such as viacolors, patterns and/or other visual indications. Furthermore, in atleast some embodiments, the displayed elements of the GUI include one ormore visual target position indicators that are displayed at locationsin the image corresponding to real-world 3D positions at which imagesare to be acquired and saved—such displayed visual target positionindicators may include, for example, one or more indicators for acurrent image to be acquired and saved, one or more indicators for anext image to be acquired and saved after the current image, etc., andoptionally to include indicators for all target positions that are knownat a time of display and visible in the current image being displayed.It will be appreciated that as a mobile device's position changes andthe corresponding images visible from the device's imaging sensorchange, the displayed GUI will similarly be updated, even if no imagesare acquired and saved during those changes. Furthermore, if images arenot successfully acquired and saved for one or more missed targetpositions, the displayed GUI may further generate and displayinstructions related to correcting the situation, such as for the userto change the direction of rotation (go backwards) to reacquire and saveimages for such missed target positions—such missed target positions maybe caused by various factors, such as the mobile device position notsuccessfully matching the target position (e.g., with respect to one ormore of the six degrees of rotation for the mobile device), the mobiledevice being turned too fast (to cause a blurry image) or otherwise notacquiring an image of sufficient quality (e.g., lighting is too low, anintervening objects blocks some or all of the building interior to beacquired, etc.). In some embodiments, if the mobile device is pivotedaround a horizontal axis such that the right and left sides of thedevice are at different heights (resulting in images that skewed withrespect to level), the image and visual target position indicators willbe displayed in such a skewed manner while the visual current positionindicator is in the image center and level—in other embodiments in sucha situation, the image and visual target position indicators will beadjusted to be shown as being level, while the location and/or skew ofthe mobile device's visual current position indicator may be changed onthe displayed image. In yet other embodiments, no visual target positionindicators may be displayed. Additional details are included belowrelated to displayed GUIs that may be used in various embodiments, suchas to display determined acquisition device 3D position (e.g.,orientation), including in the examples discussed with respect to FIGS.2A-2I.

The described techniques provide various benefits in variousembodiments, including to acquire images from a viewing location withgreater accuracy and with greater speed relative to previously existingtechniques, based at least in part on acquiring images that are atdefined positions (e.g., orientations) from viewing locations and thatmay satisfy additional defined criteria, including via use of handheldmobile devices that provide various benefits over other imageacquisition devices that are fixed at a particular height and/orlocation (e.g., benefits with respect to dynamic correction of imageacquisition via instructions to a human user holding the handheld mobiledevice, device availability and cost, etc.). Such described techniquesprovide further benefits in using such acquired images to generatecorresponding mapping information for buildings or other locations atwhich the images are acquired, including enabling improved automatednavigation of a building by mobile devices (e.g., semi-autonomous orfully-autonomous vehicles), so as to significantly reduce theircomputing power and time used to attempt to otherwise learn a building'slayout, as well as to provide improved GUIs in which other end users maymore accurately and quickly obtain information about a building'sinterior (e.g., for use in navigating that interior). Various otherbenefits are also provided by the described techniques, some of whichare further described elsewhere herein.

For illustrative purposes, some embodiments are described below in whichspecific types of information are acquired, used and/or presented inspecific ways for specific types of structures and by using specifictypes of devices—however, it will be understood that the describedtechniques may be used in other manners in other embodiments, and thatthe invention is thus not limited to the exemplary details provided. Asone non-exclusive example, while specific types of GUIs are used inspecific manners in specific examples, it will be appreciated that othertypes of GUIs may be used in the manners discussed herein. In addition,while GUIs are generated using specific types of data in specificexamples, including data from IMU sensors on a smartphone or othermobile device, it will be appreciated that the same or similar GUIs maybe generated and used in other manners in other embodiments, includingbased on other types of data (e.g., from a camera on a smartphone orother mobile device). In addition, the term “building” refers herein toany partially or fully enclosed structure, typically but not necessarilyencompassing one or more rooms that visually or otherwise divide theinterior space of the structure—non-limiting examples of such buildingsinclude houses, apartment buildings or individual apartments therein,condominiums, office buildings, commercial buildings or other wholesaleand retail structures (e.g., shopping malls, department stores,warehouses, etc.), etc. The term “acquire” or “capture” as used hereinwith reference to a building interior, viewing location, or otherlocation (unless context clearly indicates otherwise) may refer to anyrecording, storage, or logging of media, sensor data, and/or otherinformation related to spatial and/or visual characteristics of thebuilding interior or subsets thereof, such as by a recording device orby another device that receives information from the recording device.As used herein, the term “panorama image” refers to any visualrepresentation that is based on, includes or is separable into multiplediscrete component images originating from a substantially similarphysical location in different directions and that depicts a largerfield of view than any of the discrete component images depictindividually, including images with a sufficiently wide-angle view froma physical location to include angles beyond that perceivable from aperson's gaze in a single direction. The term “sequence” of viewinglocations, as used herein, refers generally to two or more viewinglocations that are each visited at least once in a corresponding order,whether or not other non-viewing locations are visited between them, andwhether or not the visits to the viewing locations occur during a singlecontinuous period of time or at multiple different time periods. Inaddition, various details are provided in the drawings and text forexemplary purposes, but are not intended to limit the scope of theinvention. For example, sizes and relative positions of elements in thedrawings are not necessarily drawn to scale, with some details omittedand/or provided with greater prominence (e.g., via size and positioning)to enhance legibility and/or clarity. Furthermore, identical referencenumbers may be used in the drawings to identify similar elements oracts.

FIG. 1A is an example block diagram of various computing devices andsystems that may participate in the described techniques in someembodiments. In particular, one or more panorama images 165 may beindividually generated and then inter-connected (or “linked”) undercontrol of an ICA system 160 that is executing on one or more servercomputing systems 180 in this example, such as with respect to one ormore buildings or other structures. In particular, in each such buildingor other structure, one or more users (not shown) may carry one or moremobile devices, such as a user mobile computing device 105, to multipleviewing locations within the building or other structure, and usefunctionality of the ICA system to automatically control at least someof the acquisition of panorama images (or other information) at eachsuch viewing location—as part of the automated operations of the ICAsystem, it may obtain various sensor data (e.g., IMU sensor data) from auser mobile device, and analyze the sensor data to monitor and controlthe image acquisition process for that user mobile device, including togenerate and modify a GUI to guide the user actions during the imageacquisition process (e.g., to control the mobile device's position in 3Dspace, such as with respect to geometric orientation and optionallylocation). In some embodiments, a mobile device 105 may include abrowser 162 and/or an ICA application 155 specific to the ICA systemthat execute in memory 152 of the device 105 by processor 132 andinteract over one or more networks 170 with a remote executing ICAsystem 160, while in other embodiments some or all such mobile devices105 may instead locally execute a copy of some or all of the ICA system(not shown in FIG. 1A) in their local memory 152 that automaticallycontrols some or all of the image acquisition activities for that device105.

In operation, a copy of the ICA system may obtain information from oneor more hardware sensors on a particular mobile device, such as tointeract with one or more sensor modules 148 on mobile device 105 thatinclude a gyroscope 148 a, accelerometer 148 b and compass 148 c in thisexample (e.g., as part of one or more IMU units, not shown separately,on the mobile device), and optionally a GPS (or Global PositioningSystem) sensor or other position determination sensor (not shown in thisexample). The ICA system may then analyze the information to determinethe mobile device's 3D geometric orientation, and in some cases 3Dlocation (e.g., if combined with additional information, such as from acamera or other imaging sensor of the imaging system 135), and use suchdevice position information to control further image acquisition. Aspart of controlling the further acquisition, the ICA system may obtaininformation about at least one next defined target position (orientationand optionally location) from the current viewing location for which toacquire and save one or more next images, and generate a correspondingGUI (optionally by updating a previously generated and displayed GUI)that is shown on the display system 142 of the mobile device 105. Asdiscussed in greater detail elsewhere herein, the GUI may be overlaid onan image that is currently visible from the imaging system 135, such aswith a device current position indicator displayed in the center of theimage (or optionally other location) to represent the current positionof the mobile device, and to allow the user to manipulate the positionof the mobile device so that the displayed current position indicatormatches one or more displayed visual target indicators in the displayedGUI of the next target position at which the next image is to beacquired. In at least some embodiments, when the current and targetpositions match, the ICA system may further automatically acquire andsave one or more images using the imaging system 135 from that position,while in other embodiments the system may note one or more currentframes for that position if video is being continuously acquired andsaved. Various other types of information may also be displayed as partof such a GUI and/or otherwise presented by the mobile device (e.g.,sounds via speakers), as discussed further below, and FIGS. 2A-2Iillustrate further details about the automated control of the imageacquisition activities, as discussed further below. In addition, FIG. 1Bshows one example of acquiring such panorama images for a particularhouse, and FIGS. 1C-1D provide further examples of linking such panoramaimages, as is also discussed further below.

An MIGM (Mapping Information Generation Manager) system 140 is furtherillustrated in FIG. 1A as executing on one or more server computingsystems to generate and provide building floor maps 145 and/or othermapping-related information (not shown) based on use of the linkedpanorama images 165 and optionally associated metadata about theiracquisition and linking. In some embodiments, the ICA system 160 andMIGM system 140 may execute on the same server computing system(s), suchas if both systems are operated by a single entity or are otherwiseexecuted in coordination with each other (e.g., with some or allfunctionality of both systems integrated together into a larger system),while in other embodiments the MIGM system may instead obtain linkedpanorama images (or other information) from one or more external sourcesand optionally store them locally (not shown) with the MIGM system forfurther analysis and use. In yet other embodiments, the ICA system mayoperate without interactions with such an MIGM system or without use ofits functionality.

One or more end users (not shown, and in at least some cases differentfrom the image acquiring user(s) who are using the mobile devices 105)of one or more client computing devices 175 may further interact overone or more computer networks 170 with the MIGM system 140 and/or theICA system 160, such as to obtain, display and interact with a generatedfloor map and/or one or more associated linked panorama images (e.g., tochange between a floor map view and a view of a particular panoramaimage at a viewing location within or near the floor map; to change thehorizontal and/or vertical viewing direction from which a correspondingview of a panorama image is displayed, such as to determine a portion ofa panorama image in a 3D spherical coordinate system to which a currentuser viewing direction is directed, and to render a corresponding planarimage that illustrates that portion of the panorama image without thecurvature or other distortions present in the original panorama image;etc.). In addition, while not illustrated in FIG. 1A, in someembodiments the client computing devices 175 (or other devices, notshown), may receive and use generated floor maps and/or other generatedmapping-related information in additional manners, such as to control orassist automated navigation activities by those devices (e.g., byautonomous vehicles or other devices), whether instead of or in additionto display of the generated information.

In the depicted computing environment of FIG. 1A, the network 170 may beone or more publicly accessible linked networks, possibly operated byvarious distinct parties, such as the Internet. In otherimplementations, the network 170 may have other forms. For example, thenetwork 170 may instead be a private network, such as a corporate oruniversity network that is wholly or partially inaccessible tonon-privileged users. In still other implementations, the network 170may include both private and public networks, with one or more of theprivate networks having access to and/or from one or more of the publicnetworks. Furthermore, the network 170 may include various types ofwired and/or wireless networks in various situations. In addition, theuser mobile computing devices may include other components and storedinformation (e.g., acquired images, image acquisition metadata, linkinginformation, etc.) that are not illustrated, and the client computingdevices 175 and server computing systems 180 may similarly includevarious hardware components and stored information, as discussed ingreater detail below with respect to FIG. 3 .

As part of the acquisition and generation of multiple panorama images atmultiple associated viewing locations (e.g., in multiple rooms or otherlocations within a building or other structure and optionally aroundsome or all of the exterior of the building or other structure), such asfor use in generating and providing a representation of an interior ofthe building or other structure, ICA system 160 may perform furtherautomated operations involved in such activities. For example, in atleast some such embodiments, such techniques may include using one ormore mobile devices 105 (e.g., a smart phone held by a user, a cameraheld by or mounted on a user or the user's clothing, etc.) to capturevisual data from a sequence of multiple viewing locations (e.g., videocaptured continuously at each viewing location while a mobile device isrotated for some or all of a full 360 degree rotation at that viewinglocation, a series of individual images acquired in a non-continuousmanner, etc.) within multiple rooms of a house (or other building), andto further capture data linking the multiple viewing locations, butwithout having distances between the viewing locations being measured orhaving other measured depth information to objects in an environmentaround the viewing locations (e.g., without using any depth-sensingsensors separate from the camera). After the viewing locations' videosand linking information are captured, the techniques may includeanalyzing video captured at each viewing location to create a panoramaimage from that viewing location that has visual data in multipledirections (e.g., a 360 degree panorama around a vertical axis),analyzing information to determine relative positions/directions betweeneach of two or more viewing locations, creating inter-panoramapositional/directional links in the panoramas to each of one or moreother panoramas based on such determined positions/directions, and thenproviding information to display or otherwise present multiple linkedpanorama images for the various viewing locations within the house.Additional details related to embodiments of a system providing at leastsome such functionality of an ICA system are included in each of thefollowing: co-pending U.S. Non-Provisional patent application Ser. No.15/649,434, filed Jul. 13, 2017 and entitled “Connecting And UsingBuilding Interior Data Acquired From Mobile Devices” (which includesdisclosure of an example BICA system that is generally directed toobtaining and using panorama images from within one or more buildings orother structures); U.S. Non-Provisional patent application Ser. No.15/950,881, filed Apr. 11, 2018 and entitled “Presenting ImageTransition Sequences Between Viewing Locations” (which includesdisclosure of an example ICA system that is generally directed toobtaining and using panorama images from within one or more buildings orother structures, as well as an example ILTM system that is generallydirected to determining and/or presenting transition sequences betweendifferent panorama images or other images); and U.S. Non-Provisionalpatent application Ser. No. 16/190,162, filed Nov. 14, 2018 and entitled“Automated Mapping Information Generation From Inter-Connected Images”(which includes disclosure of an example ICA system that is generallydirected to obtaining and using panorama images from within one or morebuildings or other structures, as well as an example FMGM system that isgenerally directed to determining and/or presenting floor maps and/orother representations of buildings or other structures using multiplecaptured panorama images or other images for those structures); each ofwhich is incorporated herein by reference in its entirety.

FIG. 1B depicts a block diagram of an exemplary building interiorenvironment in which linked panorama images will be generated andsubsequently used. In particular, FIG. 1B includes a building 198 withan interior to be captured at least in part via multiple panoramaimages, such as by a user (not shown) carrying a mobile device 185 withimage acquisition capabilities as it is moved through the buildinginterior along a travel path 115 to a sequence of multiple viewinglocations 210. An embodiment of the ICA system (e.g., ICA system 160 onserver computing system(s) 180, a copy of some or all of the ICA systemexecuting on the user's mobile device, etc.) may automatically performor assist in the capturing of the data representing the buildinginterior, as well as further analyze the captured data to generatelinked panorama images providing a visual representation of the buildinginterior. While the mobile device of the user may include varioushardware components, such as a camera, one or more hardware sensors(e.g., a gyroscope, an accelerometer, a compass, etc., such as part ofone or more IMUs of the mobile device; an altimeter; light detector;etc.), a GPS receiver, one or more hardware processors, memory, adisplay, a microphone, etc., the mobile device may not in at least someembodiments have access to or use equipment to measure the depth ofobjects in the building relative to a location of the mobile device,such that relationships between different panorama images and theirviewing locations may be determined in part or in whole based onmatching features in different images and/or by using information fromother of the listed hardware components, but without using any data fromany such depth sensors. In addition, while directional indicator 109 isprovided for reference of the reader, the mobile device and/or ICAsystem may not use such absolute directional information in at leastsome embodiments, such as to instead determine relative directions anddistances between panorama images 210 without regard to actualgeographical positions or directions.

In operation, a user associated with the mobile device arrives at afirst viewing location 210A within a first room of the building interior(in this example, an entryway from an external door 190 to the livingroom), and captures a view of a portion of the building interior that isvisible from that viewing location 210A (e.g., some or all of the firstroom, and optionally small portions of one or more other adjacent ornearby rooms, such as through doors, halls, stairs or other connectingpassages from the first room) as the mobile device is rotated around avertical axis at the first viewing location (e.g., with the user turninghis or her body in a circle while holding the mobile device stationaryrelative to the user's body). The view capture may be performed byrecording a video and/or taking a succession of images, and may includea number of objects or other features (e.g., structural details) thatmay be visible in images (e.g., video frames) captured from the viewinglocation—in the example of FIG. 1B, such objects or other features maygenerally include the doorways 190 and 197 (e.g., with swinging and/orsliding doors), windows 196, corners or edges 195 (including corner195-1 in the northwest corner of the building 198, corner 195-2 in thenortheast corner of the first room, and other corners or edges that arenot shown, including between walls and floors or ceilings), furniture191-193 (e.g., a couch 191; chairs 192, such as 192-1 and 192-2; tables193, such as 193-1 and 193-2; etc.), pictures or paintings ortelevisions or other objects 194 (such as 194-1 and 194-2) hung onwalls, light fixtures (such as light fixture 130 of FIGS. 2A-2I), etc.The user may also optionally provide a textual or auditory identifier tobe associated with a viewing location, such as “entry” for viewinglocation 210A or “living room” for viewing location 210B, while in otherembodiments the ICA system may automatically generate such identifiers(e.g., by automatically analyzing video and/or other recordedinformation for a building to perform a corresponding automateddetermination, such as by using machine learning) or the identifiers maynot be used.

After the first viewing location 210A has been adequately captured(e.g., by a full rotation of the mobile device), the user may proceed toa next viewing location (such as viewing location 210B), optionallyrecording video and/or other data from the hardware components (e.g.,from one or more IMUs, from the camera, etc.) during movement betweenthe viewing locations along a travel path 115. At the next viewinglocation, the user may similarly use the mobile device to capture one ormore images from that viewing location. This process may repeat fromsome or all rooms of the building (preferably all rooms) and optionallyexternal to the building, as illustrated for viewing locations210C-210J. The acquired video and/or other images for each viewinglocation are further analyzed to generate a panorama image for each ofviewing locations 210A-210J, including in some embodiments to matchobjects and other features in different images. Subsequent furtherprocessing and analysis may be performed in order to ‘link’ at leastsome of the panoramas together with lines 215 between them, as discussedfurther with respect to FIGS. 1C-1D, such as to determine relativepositional information between pairs of viewing locations that arevisible to each other and, to store corresponding inter-panorama links(e.g., links in directions 215-AB, 215-BC and 215-AC between viewinglocations A and B, B and C, and A and C, respectively), and in someembodiments and situations to further link at least some viewinglocations that are not visible to each other (e.g., between viewinglocations 210B and 210E).

FIGS. 1C and 1D provide further details regarding one example ofperforming linking operations using panorama images such as from viewinglocations 210 of FIG. 1B, including to determine relative positionalinformation between the viewing locations for use in inter-connectingpanorama images or other visual information corresponding to thoseviewing locations. While the example of FIGS. 1C and 1D uses informationabout a travel path that the user takes between viewing locations toperform linking operations between panorama images for those viewinglocations, linking operations between panorama images may be performedin part or in whole using other techniques in other embodiments, such asby identifying the same features in different panorama images that haveoverlapping fields of view (e.g., for different panorama images in thesame room) and by using the relative locations of those features in thedifferent images to determine relative position information between theviewing locations of the panorama images.

In particular, FIG. 1C provides additional information 103, includingabout portions of the path 115 ab and 115 bc that reflect the usermoving from viewing location 210A to viewing location 210B, andsubsequently from viewing location 210B to 210C, respectively. It willbe appreciated that the order of obtaining such linking information mayvary, such as if the user instead started at viewing location 210B andcaptured linking information as he or she traveled along path 115 bc toviewing location 210C, and later proceeded from viewing location 210A toviewing location 210B along travel path 115 ab with correspondinglinking information captured (optionally after moving from viewinglocation 210C to 210A without capturing linking information). In thisexample, FIG. 1C illustrates that the user departs from the viewinglocation 210A at a point 137 in a direction that is just west of duenorth (as previously indicated with respect to directional indicator 109of FIG. 1B), proceeding in a primarily northward manner forapproximately a first half of the travel path 115 ab, and then beginningto curve in a more easterly direction until arriving at an incomingpoint 138 to viewing location 210B in a direction that is mostlyeastward and a little northward. In order to determine the departuredirection from point 137 more specifically, including relative to thedirection 120A at which the video acquisition previously began forviewing location 210A (and at which the resulting panorama imagebegins), initial video information captured as the user travels alongtravel path 115 ab may be compared to the frames of the panorama imagefor viewing location 210A in order to identify matching frames/images.In particular, by matching one or more best frames in that panoramaimage that correspond to the information in the initial one or morevideo frames/images taken as the user departs from point 137, thedeparture direction from point 137 may be matched to the viewingdirection for acquiring those matching panorama images—while notillustrated, the resulting determination may correspond to a particulardegree of rotation from the starting direction 120A to the one or morematching frames/images of the panorama image for that departuredirection. In a similar manner, in order to determine the arrivaldirection at point 138 more specifically, including relative to thedirection 120B at which the video acquisition began for viewing location210B (and at which the resulting panorama image begins), final videoinformation captured as the user travels along travel path 115 ab may becompared to the frames of the panorama image for viewing location 210Bin order to identify matching frames/images, and in particular toframes/images in direction 139 (opposite to the side of viewing location210B at which the user arrives).

While such departure direction and arrival direction would allow theactual relative direction 215-AB between the viewing locations 210A and210B to be determined if the travel path 115 ab was substantiallystraight, that is not the case in this example—instead, in order todetermine the direction 215-AB, acceleration data captured as part ofthe linking information for the travel path 115 ab is analyzed toidentify user velocity and location along the travel path 115 ab, inorder to model the resulting relative locations of the travel pathbetween starting point 137 and arrival point 138. In this example, theacceleration data acquired for the north-south direction (e.g., from oneor more IMU units in a mobile device carried by the user) indicates thatthere is an initial significant acceleration spike in the northerlydirection as the user began moving, which then drops to near zero as theuser maintains a constant velocity in a generally northern directionalong the middle portion of the travel path 115 ab, and then begins alonger but less sharp acceleration in the southerly direction as theuser curves to a primarily easterly direction toward viewing location210B and decelerates at arrival. The acceleration data may be integratedto determine corresponding north-south velocity information, and thenfurther integrated to determine location information for each datapoint. By combining the determined velocity and location information, anamount of north-south movement by the user along travel path 115 ab maybe determined, corresponding to an aggregate amount of north-southdistance traveled between viewing locations 210A and 210B. In a similarmanner, acceleration and velocity information may be determined in aneast-west direction for the travel path 115 ab as the user moves alongthe travel path, with the resulting double integration in velocity andlocation data providing an aggregate amount of east-west distance thatthe user travels along the travel path 115 ab. By combining theaggregate north-south and east-west distances (and assuming in thisexample that no height change occurred, although height information maybe obtained and analyzed in a similar manner in some embodiments) withthe determined departure and arrival information, a total distancetraveled between viewing locations 210A and 210B in a correspondingdirection 215-AB is determined (with direction 215-AB being a two-waydirection in this example, from viewing location 210A to 210B and fromviewing location 210B to 210A).

Based on a similar analysis of departing direction from viewing location210B, arrival direction at viewing location 210C, and interveningvelocity and location for some or all data points for which accelerationdata is captured along the travel path 115 bc, the user's movement fortravel path 115 bc may be modeled, and resulting direction 215-BC andcorresponding distance between viewing locations 210B and 210C may bedetermined. As a result, inter-panorama link 225B-C may be determined ina direction 215-BC to viewing location 210C, with correspondinginformation included in the panorama image generated at viewing location210B, and inter-panorama link 225C-B may similarly be determined indirection 215-BC to viewing location 210B, with correspondinginformation included in the panorama generated at viewing location 210C.Similarly, inter-panorama link 225A-B from viewing location 210A to 210Bmay be determined in a direction 215-AB to viewing location 210B from210A, with corresponding information included in the panorama imagegenerated at viewing location 210A, and inter-panorama link 225B-A maysimilarly be determined in direction 215-AV from viewing location 210Bto viewing location 210A, with corresponding information included in thepanorama generated at viewing location 210B.

Despite the lack of linking information captured between viewinglocations 210A and 210C (e.g., because the user did not travel along apath between those viewing locations, because linking information wasnot captured as a user did travel along such a path, etc.), information103 further illustrates an example of direction 226 that may optionallybe determined between viewing locations 210A and 210C based at least inpart on the analysis of linking information for travel paths 115 ab and115 bc (and with corresponding inter-panorama links 225A-C and 225C-Abeing generated in direction 226 and included in the panorama images forviewing locations 210A and 210C, respectively). In particular, even ifan absolute location of viewing locations 210A, 210B and 210C are notknown from the analysis of the linking information for travel paths 115ab and 115 bc, relative locations of those viewing locations may bedetermined in a manner discussed above, including estimated distancesand directions between viewing locations 210A and 210B and betweenviewing locations 210B and 210C. In this manner, the third side of theresulting triangle having determined lines 215-AB and 215-BC may bedetermined to be line 226 using geometrical analysis, despite the lackof direct linking information between viewing locations 210A and 210C.It will be further noted that the analysis performed with respect totravel paths 115 ab and 115 bc, as well as the estimation of directionand distance corresponding to 226, may be performed in this exampleregardless of whether or not viewing locations 210A, 210B and/or 210Care visible to each other—in particular, even if the three viewinglocations are in different rooms and/or are obscured from each other bywalls (or by other structures or impediments), the analysis of thelinking information may be used to determine the relative locationsdiscussed above (including directions and distances) for the variousviewing locations. It will be appreciated that the techniquesillustrated with respect to FIGS. 1C and 1D may be continued for allviewing locations in building 198, resulting in a set of linked panoramaimages corresponding to viewing locations 210A-J, or otherwise in othersimilar buildings or other structures.

FIG. 1D depicts a block diagram of the building interior environment inwhich the panorama images have been generated and linked. In particular,FIG. 1D is similar to FIG. 1B, but lacks details about the path 115 anduser mobile device 185 (as well as exterior viewing locations 210I-J),while adding information about inter-panorama links in two-waydirections 215, which include illustrated links between viewinglocations 215A and 215B in direction 215-AB, between viewing locations215B and 215C in direction 215-BC, and between viewing locations 215Aand 215C in direction 215-AC. FIG. 1D further illustrates thatinter-panorama links may be determined and used in some embodiments andsituations to further link at least some viewing locations that are notvisible to each other (e.g., the link in direction 215-BE betweenviewing locations 210B and 210E).

Various details are provided with respect to FIGS. 1A-1D, but it will beappreciated that the provided details are non-exclusive examplesincluded for illustrative purposes, and other embodiments may beperformed in other manners without some or all such details.

FIGS. 2A-2I illustrate examples of automatically analyzing and usingsensor data on a handheld mobile device that is acquiring images from abuilding interior in order to control the image acquisition, such asbased on the building 198 and inter-connected panorama images forviewing locations 210 discussed in FIGS. 1B-1D. In these examples,various details are discussed with respect to the panorama imageacquired at viewing location 210B in the living room 229 a of theillustrated building—it will be appreciated that similar analysis may beperformed for that same room by using the panorama image information forviewing locations 210A and 210C, and for the other rooms of thebuilding. In this example, the southeast room of the building does nothave any viewing locations within or closely adjacent to the room, andthus an analysis may not be performed for it with respect to suchin-room viewing location position, although information from otherviewing locations with visibility into that room (e.g., viewinglocations 210G and 210H) may be used at least in part for other types ofinformation acquired from analysis of panorama images. In addition, forpurposes of this example, small areas such as closets and alcoves/nooksare not analyzed as separate rooms, but are instead treated as part ofthe larger containing room (optionally as unmapped space within theroom), although in other embodiments such small areas could instead beseparately represented (including to optionally have one or more viewinglocations located within them). Similarly, while the east-west hallwayis modeled as a separate room in this example, in other embodiments suchhallways may instead be treated as part of one or more rooms connectingto the hallway, or instead the hallway may be treated as a connectingpassage between rooms rather than as a separate room.

In particular, FIG. 2A illustrates an image 250 a that may be visible ona display of a user mobile device (or other device) that is capturedwhen a user is standing at viewing location 210B of room 229 a ofbuilding 198 in starting direction 120B, as discussed in greater detailwith respect to FIGS. 1B-1D. In this example, the user has not yet begunto acquire a panorama image from the viewing location 210B, but is aboutto initiate that process. FIG. 2A illustrates various objects in theroom 229 a that are visible in the image 250 a, including chair 192-1,border 195-2, picture or painting 194-1, windows 196, and light fixtures130 a and 130 b.

FIG. 2B continues the example of FIG. 2A, and illustrates informationfrom a displayed GUI overlaid on the image 250 a of FIG. 2A. Inparticular, and is discussed in greater detail elsewhere herein, the ICAsystem may use information from the hardware sensors of the handhelduser mobile device to determine a current position in 3D space(geometric orientation and optionally location) of the mobile device asit is held by the user, and use that information as part of thedisplayed GUI. In this example, the displayed GUI includes a displayedcurrent position indicator 270 that represents the current position ofthe mobile device, which in this example is centered in the middle ofthe image 250 b. In addition, the display GUI includes a number ofdisplayed target position indicators 280 in this example to indicatetarget positions from viewing location 210B at which to acquire imagesfor use in generation of a panorama image representing viewing location210B, although such visual indicators may not be included in otherembodiments (e.g., in embodiments in which continuous video is capturedrather than a series of separate images)—in this example, the displayedtarget position indicators include a current target position indicator280 a, a next target position indicator 280 b, and additional futuretarget position indicators as are illustrated. The displayed GUI in thisexample further includes a directional indicator 285 b to provide avisual indication of a direction in which the user is to rotate or turnat viewing location 210B, as well as a textural overlay 275 b in thisexample to provide instructions regarding beginning the capture of thepanorama image from the viewing location.

The target positions at which to acquire and save images from a viewinglocation may be determined in various manners in various embodiments.For example, in some embodiments a defined quantity N of images isselected and evenly spaced in a 360° circle around a viewing location(e.g., to correspond to every M degrees, where M is 360 divided by N) ata constant angle of tilt, swivel and pivot relative to the user'sturning body (e.g., by maintaining the handheld mobile device in alevel, non-changing position relative to the user's body while the bodyturns). In other embodiments, the target positions may be determinedrelative to one or previous positions at which images are acquired andsaved, such as based on an amount of rotation and/or time since a lastsaved image, an amount of overlap in content with one or more previoussaved images, etc. In addition, in some embodiments the target positionsare determined to be continuous or substantially continuous, such aswhen video is being acquired at a rotation rate that matches or is belowa defined threshold, and some or all acquired video frames correspond totarget positions. Furthermore, while the target positions are in someembodiments acquired at a single angle of tilt, swivel and pivotrelative to the user's turning body (e.g., straight outward andperpendicular to a line bisecting the user's body from head to feet), inother embodiments the target positions may be selected to vary in one ormore such rotational measures, and potentially to be acquired inmultiple user rotations at a viewing location—as one example, in orderto generate a panorama that allows an end user to tilt up to see theceiling and/or down the see the floor (e.g., for a panorama image thatincludes data for a complete or largely complete sphere from a viewinglocation), target positions may include some with a tilt angle thatcorresponds to straight outward from the user's turning body (i.e.,perpendicular to a line bisecting the user's body from head to feet),some with a tilt angle downward (e.g., 45° down relative to outward),and some with a tilt angle upward (e.g., 45° up relative to outward). Inother embodiments, the target positions may be determined in part or inwhole based on user preferences or other user configuration performed(e.g., before the image acquisition begins at a viewing location). Itwill be appreciated that target positions may be selected in othermanners in other embodiments.

In addition, while the example of FIG. 2B (and the continuing examplesof FIGS. 2C-2I) illustrate capturing images in the visual lightspectrum, it will be appreciated that other embodiments may captureother types of information, whether instead of or in addition to suchvisual light images. Examples of such other types of information thatmay be acquired and saved, and optionally included in the panorama imageto be generated or otherwise associated with it, including light in oneor more other ranges (e.g., infrared, ultraviolet, etc.), other types ofelectrical signals and/or magnetic fields (e.g., radiation, cell phonesignals, Wi-Fi signals, etc.), audio information from the surroundingenvironments, verbal and/or textual annotations (e.g., descriptions ofthe viewing location and/or objects in the surrounding environment),etc. Furthermore, when target position indicators are visually includedin a displayed GUI (whether for target positions whose images and/orother information has already been acquired and saved, and/or for targetpositions whose images and/or other information are to be saved in thefuture), they may further be altered in some embodiments (e.g., withrespect to shape, size, visual appearance, type of element, etc.) toindicate one or more types of metadata associated with those targetpositions, such as the type of information acquired, recency ofacquisition, quality of acquired information, etc.

FIG. 2C continues the examples of FIGS. 2A-2B, and illustrates an image250 c that represents a short time after that of FIG. 2B, and in whichthe panorama image capture process has begun. In particular, in thisexample the visual current position indicator 270 of the mobile devicecontinues to be centered in the image 250 c, but as the user has begunto turn at viewing location 210B, additional portions of the room to theright have become visible, while portions of the room to the left (e.g.,light fixture 130 b) are no longer visible. In addition, as the userturns, the ICA system continues to monitor data from the hardwaresensors of the mobile device in a real time manner, including todetermine current position (geometric orientation and optionallylocation) of the mobile device as the user turns. In this example,additional visual past target position indicators 290 have been added tothe left of the current position indicator 270, which representindications of past target positions at which images should have alreadybeen captured, and which in this example have been successfullycompleted. Accordingly, the current target position indicator 280 a andnext target position indicator 280 b have been moved relative to theroom interior to represent the changing next position at which toacquire additional images, and the turn direction indicator 285 c hassimilarly been moved. In this example, the textual indicator 275 c hasalso been updated, although in other embodiments the textual indicatormay not change or may no longer be displayed in this situation. In thisexample, the past target position indicators 290 of acquired imagesfurther may use visual indications that indicate a success and/orquality of the images acquired and saved from those positions, such asto use a particular color to indicate a full success, and optionally oneor more other colors to indicate a lack of success or degrees of successat the capture. In this example, the successful visual acquisitionindication is a cross hatch pattern shown in the past target positionindicators 290, although it will be appreciated that other visualindications may be used in other embodiments, and that in someembodiments such visual indications of success or lack of success maynot be shown, and further than in other embodiments no such past targetposition indicators 290 (also referred to herein as a “trail”) may beillustrated.

FIG. 2D continues the examples of FIGS. 2A-2C, and corresponds to aperiod of time shortly after that illustrated in FIG. 2C. While theexample of FIG. 2C previously indicated a continuing successful captureof images as the user rotated, based on the user maintaining a correctposition of the mobile device as he or she turned, FIG. 2D illustratesan alternative situation in which the orientation of the mobile deviceis no longer correct, and corresponds to the mobile device being pivotedside-to-side (with the right side of the mobile device higher than theleft side) such that it is not level with the room, and such that itscurrent position indicator 270 no longer matches the indicators 280 forthe target positions to acquire. In this example, the current positionindicator 270 of the mobile device continues to be displayed in thecenter of the new image 250 d and in the same level orientation relativeto the borders of the image, but the actual room and its contents areshown in a skewed manner to correspond to the image captured by themobile device due to its rotation. Accordingly, the displayed GUI hasbeen updated in multiple manners in this example to indicate the lack ofsuccessful capture of one or more images for current and/or past targetpositions due to the incorrect rotation of the user device—inparticular, the ICA system acquires and uses data from hardware sensorsof the handheld mobile device as the user turns to determine whether ornot to capture images for the target positions, such as to not capturethe image for the past target position 292 in this example due to theskewing of the handheld mobile device and/or other problems with thepositioning and movement of the mobile device for that past targetposition, and to capture images at other past target positions 291 inthis example but to associate them with lower quality (e.g., topotentially replace them with higher quality images if they becomeavailable, and/or to not use those images or to otherwise give themlower weight or other influence when later using the captured images togenerate additional mapping-related information). Accordingly, thetextual indicator 275 d has been updated to indicate to the user to gobackwards and recapture one or more images that were not successfullycaptured at one or more past target positions, which in this examplecorresponds to at least the past target position 292, as is shown withempty contents and no pattern to indicate a lack of successful imageacquisition and saving, and in other embodiments could be illustratedwith a particular color or in other visual manners. The turn indicator285 d that is illustrated similarly indicates to the user to return inthe opposite direction of the previous user turn to arrive back at thetarget position 292 that needs to be recaptured. In this example, twoadditional past target positions have indicators 291 that are shown witha different fill pattern (and in other embodiments could have adifferent color or other visual indication) to indicate partial success,such as to indicate that the images are not ideal but meet a minimumthreshold of quality for use in the subsequent generation of thepanorama image, although in other embodiments such non-ideal capturedimages may similarly be recaptured. Alternatively, such non-ideal orfailed image acquisitions for past target positions may not berecaptured in this manner, or at all, in other embodiments—as oneexample, a user may instead be requested to do multiple rotations of theroom from the viewing location 210B, such that a second rotation mayprovide sufficient images at one or more such target positions 292and/or 291, while in other embodiments the generation of the panoramaimage may be possible from other nearby successful image orientations290. In addition, the further rotation of the user illustrated in FIG.2E has now brought part of table 193-1 into view.

FIG. 2E continues the examples of FIGS. 2A-2D, and in this caseillustrates an image 250 e that is an alternative type of problem thatmay occur at the same location as that previously discussed with respectto FIG. 2D. In particular, the orientation of the mobile device is notpivoted in this example, but is tilted downward (with the top of themobile device farther from the user's body than the bottom of the mobiledevice) at a level such that its current position indicator 270 nolonger matches the indicators 280 for the target positions to acquire,similarly resulting in a past target position 292 that was notsuccessfully captured, and other past target positions 291 correspondingto partial success in their respective images. It will be appreciatedthat other types of problems with orientation and/or location of themobile device may similarly occur, as well as other problemscorresponding to particular images (e.g., insufficient light, userrotation that is too fast so as to cause blurring, a finger or otherobject blocking a portion of the image being captured, etc.).

FIG. 2F continues the examples of FIGS. 2A-2E, and corresponds to aperiod of time at which the user has completed a full rotation atviewing location 210B and successfully captured images at each of thedetermined target positions, as illustrated by the completed visual pasttarget position indicators 290 that are overlaid on image 250 f of FIG.2F. Since the user has completed a full rotation, the view of the roomin the direction of the image 250 f corresponds to that of the beginningimage 250 a when the panorama image capture began. In this example, theillustrated GUI has been updated to include additional textualinformation 275 f with instructions to the user to continue the process,and in particular to connect the panorama image for the viewing location210B that was just captured to a next viewing location and correspondingpanorama to be captured, such as at viewing location 210C in thisexample. It will be appreciated that other types of directions may beused in other embodiments to assist a user in moving to a next viewinglocation and/or capturing acquisition metadata along a travel path tothat next viewing location.

FIG. 2G illustrates an alternative image 250 g that corresponds to aview of a completed panorama image from viewing location 210A in adirection 215-AB toward viewing location 210B, such as after thegeneration of inter-connected panorama images for building 198 iscomplete. In this example, a dashed line 250 a is indicated to show theimage visible from viewing location 210B within the larger current image250 g (with a wider angle of view, since taken at a greater distancefrom the objects shown in FIG. 2A), although such a visual indication250 a may not be displayed to a user in other embodiments. FIG. 2Gfurther illustrates that after panorama images have been generated andlinked, the displayed GUI shown to an end user may further includevisual elements that allow movement between viewing locations and theirpanorama images, such as in this example to include a user-selectablecontrol 241 that the user may select to move to view the panorama imageacquired at viewing location 210B. The displayed GUI further includestextual information 243 that provides a textual indicator of the viewinglocation 210B, which in this example is labeled “living room.” Inaddition, FIG. 2G further illustrates information 245 that may bedisplayed in some embodiments to a user, whether after the generation ofinter-connected panorama images for building 198 is complete, and/orduring the process of capturing additional panorama images after apanorama image has been acquired at viewing location 210B and while theuser is at a different viewing location from which that viewing location210B is visible. In particular, in this example the visual indicators245 may illustrate to the user that the panorama images have beenalready acquired for the illustrated location corresponding to viewinglocation 210B, such as to provide a visual indication to the user that acorresponding portion of the building interior has already been capturedand does not need to be repeated. Alternatively, such visual indicators245 may be displayed to an end user in some embodiments to illustratethat other images are available for viewing at the illustratedpositions, whether in addition to or instead of indicators 241 and 243,and optionally in a user-selectable manner such that the user may selectone of the displayed indicators 245 and switch to a view in the viewinglocation 210B's generated panorama in that direction. In otherembodiments, such visual indicators 245, 241 and/or 243 may not beillustrated to end users (e.g., using client computing devices 175 ofFIG. 1A, and after completion of the inter-panorama connection process)and/or users during the process of capturing and generating the panoramaimages. While not illustrated in FIG. 2G, an end user may furthermanipulate the displayed panorama image view 250 g in various ways, suchas by the user dragging, scrolling, zooming or otherwise moving the viewdirection to alter the view visible from viewing location 210A'sgenerated panorama image.

FIGS. 2H and 2I continue the examples of FIGS. 2A-2G, and particularillustrate alternative examples of current position indicators that mayused to assist a user during the process of acquiring a panorama imageat a viewing location. In particular, image 250 h of FIG. 2H is similarto image 250 b of FIG. 2B from viewing location 210B. However, in FIG.2H, the displayed GUI that is overlaid on the image 250 h does notinclude the previous visual indicator 270 of the mobile device, andinstead indicates one or more alternative visual current positionindicators 271 h to assist a user in determining the position (e.g.,orientation) of the mobile device. In this example, the visual currentposition indicator 271 h includes horizontal and vertical bars displayedperpendicular to each other (such as in a manner similar to an attitudeindicator used for an airplane to show the horizon and level flight)—inthis example, the visual current position indicator 271 h-1 is similarlyshown in the center of the image, such as to correspond to the beginningof the panorama image acquisition process, with other similar visualtarget position indicators 280 also illustrated.

FIG. 2H further illustrates an example 271 h-2 of an alternative visualcurrent position indicator, and which corresponds to a situation inwhich the user is holding the mobile device at an incorrect orientation(which in this example includes the mobile device being tilted down fromthe appropriate level of the target indicators 280, being rotated to theright, and being pivoted with the top part of the mobile device fatheraway from the user than the bottom part). In this example, while theactual image of the room from the mobile device's imaging sensor wouldbe rotated and skewed to correspond to the mobile device's orientation(in a manner similar to that illustrated with respect to FIG. 2D), theimage 250 h of the room has been displayed as level, while the currentposition (orientation) indicator 271 h-2 has been moved within the imageto correspond to the incorrect orientation of the mobile device—in otherembodiments, the display of the visual indicator 271 h-2 may instead beat the center of image while the remaining part of the image isdisplayed based on what is currently visible from the camera or otherimaging sensor of the mobile device.

FIG. 2I illustrates an image 250 i that is similar to that of FIG. 2H,but in which the current position indicator(s) 271 h are not shown, andinstead a modified version of the indicator 270 of FIG. 2B is shown in asmaller version 270 i that is similarly in the center of image 250 i. Inaddition, in this example the displayed GUI further includes extendedupper and lower borders 271 i in the turning direction 285 i, such thatthe user is to keep the displayed indicator 270 i within the upper andlower boundaries as the user rotates at viewing location 210B. In thisexample, the displayed GUI further includes indicators 282 to indicatetarget positions at which the images will be acquired (instead of thetarget position indicators 280 of FIG. 2B), although such visualindicators may not be included in other embodiments, including inembodiments in which continuous video is captured rather than a seriesof separate images. It will be appreciated that the displayed GUI ofFIGS. 2B-2I may be modified in a variety of other manners that are notexplicitly illustrated here.

In addition, while not illustrated in FIGS. 2A-2I, the ICA system mayprovide other types of real-time feedback to the user of the mobiledevice in some embodiments via one or more visual guidance cues duringthe capture of images (including via continuous video) at a viewinglocation. For example, the ICA system may determine (such as based onsensor data provided by sensor modules 148) that the mobile device isrotating too quickly, and if so may provide an auditory, visual, orother appropriate notification to indicate that the user should rotatethe mobile device more slowly. As another example, the ICA applicationmay determine that the mobile device is shaking or otherwise failing toprovide high quality video (such as based on sensor data or one or moreanalyses of particular captured video frames), and if so may provide anotification to advise the user of the problem. As still anotherexample, the ICA system may provide a notification to the user if it isdetermined that a particular viewing location is unsuitable forcapturing information about the building interior, such as if the ICAsystem detects that lighting conditions or other environmental factorsfor the present viewing location are negatively affecting the recordingprocess. Furthermore, the ICA system may in certain embodiments prompt auser for information regarding one or more of the viewing locationsbeing captured, such as to provide a textual or auditory identifier tobe associated with a viewing location (e.g., “Living Room,” “Office,”“Bedroom 1” or other identifier), or to otherwise capture descriptiveinformation from the user about the room (e.g., a description ofbuilt-in features, a history of remodels, information about particularattributes of the interior space being recorded, etc.). In otherembodiments, such identifiers and/or other descriptive information maybe determined in other manners, including automatically analyzing videoand/or other recorded information for a building (e.g., using machinelearning) for the determination. In at least one embodiment, suchacquired or otherwise determined identifiers and/or other descriptiveinformation may be later incorporated in or otherwise utilized with thecaptured information for a viewing location, such as to provide atextual or auditory indication of the identifier or other descriptiveinformation during subsequent display or other presentation of thebuilding interior by the ICA system (or by another system that receivescorresponding information from the ICA system).

Various details have been provided with respect to FIGS. 2A-2I, but itwill be appreciated that the provided details are non-exclusive examplesincluded for illustrative purposes, and other embodiments may beperformed in other manners without some or all such details.

FIG. 3 is a block diagram illustrating an embodiment of one or moreserver computing systems 300 executing an implementation of an ICAsystem 340—the server computing system(s) and ICA system may beimplemented using a plurality of hardware components that formelectronic circuits suitable for and configured to, when in combinedoperation, perform at least some of the techniques described herein. Inthe illustrated embodiment, each server computing system 300 includesone or more hardware central processing units (“CPU”) or other hardwareprocessors 305, various input/output (“I/O”) components 310, storage320, and memory 330, with the illustrated I/O components including adisplay 311, a network connection 312, a computer-readable media drive313, and other I/O devices 315 (e.g., keyboards, mice or other pointingdevices, microphones, speakers, GPS receivers, etc.).

The server computing system(s) 300 and executing ICA system 340 maycommunicate with other computing systems and devices via one or morenetworks 399 (e.g., the Internet, one or more cellular telephonenetworks, etc.), such as user mobile computing devices 360 (e.g., usedto capture building interior data; used to store and provide additionalinformation related to buildings; etc.), MIGM server computing system(s)380 (e.g., on which an MIGM system executes to generate and providefloor maps and/or other related mapping information 386), user clientcomputing systems 390 (e.g., used to view linked panorama images and/orother related information; etc.), and optionally other navigable devices395 that receive and use floor maps and optionally other generatedinformation for navigation purposes (e.g., for use by semi-autonomous orfully autonomous vehicles or other devices).

In the illustrated embodiment, an embodiment of the ICA system 340executes in memory 330 in order to perform at least some of thedescribed techniques, such as by using the processor(s) 305 to executesoftware instructions of the system 340 in a manner that configures theprocessor(s) 305 and computing system 300 to perform automatedoperations that implement those described techniques. The illustratedembodiment of the ICA system may include one or more components, notshown, to each perform portions of the functionality of the ICA system,and the memory may further optionally execute one or more other programs335—as one specific example, a copy of the MIGM system may execute asone of the other programs 335 in at least some embodiments, such asinstead of or in addition to the MIGM system 389 on the MIGM servercomputing system(s) 380. The ICA system 340 may further, during itsoperation, store and/or retrieve various types of data on storage 320(e.g., in one or more databases or other data structures), such asvarious types of user information 322, linked panorama image information324 (e.g., to provide to users of client computing devices 360 fordisplay; for analysis to generate floor maps; etc.), optionallygenerated floor maps and other associated information 326 (e.g.,generated and saved 3D models, building and room dimensions for use withassociated floor plans, additional images and/or annotation information,etc.) and/or various types of optional additional information 328 (e.g.,defined target position information for image acquisition, definedthresholds to use in assessing image capture information, etc.).

Some or all of the user mobile computing devices 360 (e.g.,smartphones), client computing systems 380, client computing systems 390and other navigable devices 395 may similarly include some or all of thesame types of components illustrated for server computing system 300. Asone non-limiting example, the server computing systems 380 are eachshown to include one or more hardware CPU(s) 381, I/O components 382,storage 385, and memory 387, with an embodiment of the MIGM system 389executing within memory 387, and with mapping information 386 that isgenerated by the MIGM system being stored on storage 385. As anothernon-limiting example, the mobile client computing devices 360 are eachshown to include one or more hardware CPU(s) 361, I/O components 362,storage 365, imaging system(s) 364 with one or more imaging sensors (notshown), IMU hardware sensor(s) 369, and memory 367, with one or both ofa browser 368 and one or more client applications 368 (e.g., anapplication specific to the ICA system) executing within memory 367,such as to participate in communication with the ICA system 340, MIGMsystem 389 and/or other computing systems—in other embodiments, a copyof some or all of the ICA system may instead execute on each of some orall of the user mobile computing devices 360. While particularcomponents are not illustrated for the client computing systems 390 orother navigable devices 395, it will be appreciated that they mayinclude similar and/or additional components.

It will also be appreciated that computing system 300 and the othersystems and devices included within FIG. 3 are merely illustrative andare not intended to limit the scope of the present invention. Thesystems and/or devices may instead each include multiple interactingcomputing systems or devices, and may be connected to other devices thatare not specifically illustrated, including via Bluetooth communicationor other direct communication, through one or more networks such as theInternet, via the Web, or via one or more private networks (e.g., mobilecommunication networks, etc.). More generally, a device or othercomputing system may comprise any combination of hardware that mayinteract and perform the described types of functionality, optionallywhen programmed or otherwise configured with particular softwareinstructions and/or data structures, including without limitationdesktop or other computers (e.g., tablets, slates, etc.), databaseservers, network storage devices and other network devices, smart phonesand other cell phones, consumer electronics, wearable devices, digitalmusic player devices, handheld gaming devices, PDAs, wireless phones,Internet appliances, and various other consumer products that includeappropriate communication capabilities. In addition, the functionalityprovided by the illustrated ICA system 340 may in some embodiments bedistributed in various components, some of the described functionalityof the ICA system 340 may not be provided, and/or other additionalfunctionality may be provided.

It will also be appreciated that, while various items are illustrated asbeing stored in memory or on storage while being used, these items orportions of them may be transferred between memory and other storagedevices for purposes of memory management and data integrity.Alternatively, in other embodiments some or all of the softwarecomponents and/or systems may execute in memory on another device andcommunicate with the illustrated computing systems via inter-computercommunication. Thus, in some embodiments, some or all of the describedtechniques may be performed by hardware means that include one or moreprocessors and/or memory and/or storage when configured by one or moresoftware programs (e.g., by the ICA system 340 and/or ICA clientsoftware 369 executing on server computing systems 300 and/or mobilecomputing devices 360) and/or data structures, such as by execution ofsoftware instructions of the one or more software programs and/or bystorage of such software instructions and/or data structures, and suchas to perform algorithms as described in the flow charts and otherdisclosure herein. Furthermore, in some embodiments, some or all of thesystems and/or components may be implemented or provided in othermanners, such as by consisting of one or more means that are implementedpartially or fully in firmware and/or hardware (e.g., rather than as ameans implemented in whole or in part by software instructions thatconfigure a particular CPU or other processor), including, but notlimited to, one or more application-specific integrated circuits(ASICs), standard integrated circuits, controllers (e.g., by executingappropriate instructions, and including microcontrollers and/or embeddedcontrollers), field-programmable gate arrays (FPGAs), complexprogrammable logic devices (CPLDs), etc. Some or all of the components,systems and data structures may also be stored (e.g., as softwareinstructions or structured data) on a non-transitory computer-readablestorage mediums, such as a hard disk or flash drive or othernon-volatile storage device, volatile or non-volatile memory (e.g., RAMor flash RAM), a network storage device, or a portable media article(e.g., a DVD disk, a CD disk, an optical disk, a flash memory device,etc.) to be read by an appropriate drive or via an appropriateconnection. The systems, components and data structures may also in someembodiments be transmitted via generated data signals (e.g., as part ofa carrier wave or other analog or digital propagated signal) on avariety of computer-readable transmission mediums, includingwireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). Such computer programproducts may also take other forms in other embodiments. Accordingly,embodiments of the present disclosure may be practiced with othercomputer system configurations.

FIGS. 4A-4D illustrate an example flow diagram of an embodiment of anICA System routine 400. The routine may be performed by, for example,the ICA System 160 of FIG. 1A, the ICA System 340 of FIG. 3 , and/or theICA system described with respect to FIGS. 1B-2I and as otherwisedescribed herein, such as to acquire panorama or other images at viewinglocations within buildings or other structures in order to generate andlink panorama images for multiple such viewing locations, while usingsensor data from the acquiring mobile device to control at least some ofthe image acquisition process. While portions of the example routine 400are discussed with respect to acquiring images at particular targetpositions from a viewing location, it will be appreciated that this or asimilar routine may be used to acquire video or otherwise acquire aseries of images in a continuous or substantially continuous manner. Inaddition, while the illustrated embodiment acquires and uses informationfrom the interior of a target building, it will be appreciated thatother embodiments may perform similar techniques for other types ofdata, including for non-building structures and/or for informationexternal to one or more target buildings of interest. Furthermore, someor all of the routine may be executed on a mobile device used by a userto acquire image information, and/or by a system remote from such amobile device.

The illustrated embodiment of the routine begins at block 405, whereinstructions or information are received. At block 410, the routinedetermines whether the received instructions or information indicate toacquire data representing a building interior as part of generatingpanorama images for one or more viewing locations and optionally linkingmultiple such panorama images, and if not continues to block 490.Otherwise, the routine proceeds to block 412 to receive an indicationfrom a user of a mobile device to begin the image acquisition process ata first viewing location. After block 412, the routine proceeds to block415 in order to perform a viewing location image acquisition subroutine(with one example of such a routine illustrated in FIG. 4B, as discussedfurther below) in order to acquire for the viewing location in theinterior of the target building of interest.

After block 415 is completed and the corresponding acquired image isreceived from the subroutine and/or otherwise stored for subsequent use(including in some cases the generated panorama image for that viewinglocation), the routine continues to block 420 to determine if there aremore viewing locations to acquire, such as based on correspondinginformation provided by the user of the mobile device. If so, and whenthe user is ready to continue the process, the routine continues toblock 422 to initiate the capture of linking information (includingacceleration data) during movement of the mobile device along a travelpath away from the current viewing location and towards a next viewinglocation within the building interior. As described elsewhere herein,the captured linking information may include additional sensor data, aswell as additional video information, recorded during such movement.Initiating the capture of such linking information may be performed inresponse to an explicit indication from a user of the mobile device orbased on one or more automated analyses of information recorded from themobile device. In addition, the routine may further optionally monitorthe motion of the mobile device in some embodiments during movement tothe next viewing location, and provide one or more guidance cues to theuser regarding the motion of the mobile device, quality of the sensordata and/or video information being captured, associatedlighting/environmental conditions, advisability of capturing a nextviewing location, and any other suitable aspects of capturing thelinking information. Similarly, the routine may optionally obtainannotation and/or other information from the user regarding the travelpath, such as for later use in presentation of information regardingthat travel path or a resulting inter-panorama connection link. In block424, the routine determines that the mobile device has arrived at thenext viewing location (e.g., based on an indication from the user, basedon the forward movement of the user stopping for at least a predefinedamount of time, etc.), for use as the new current viewing location, andcontinues to block 415 in order to perform the viewing location imageacquisition subroutine.

As discussed further in FIG. 4B with respect to performing the viewinglocation image acquisition subroutine for each viewing location, theroutine begins at block 440, where it determines one or more targetpositions from the current viewing location at which to acquire images(in some cases determining all target positions, such as if they are notdependent on previous images acquired for the viewing location), andwith each target position indicating an orientation from the viewinglocation, and optionally a location translation along x, y and/or z axesrelative to the user's body or center of the viewing location. Theroutine first selects a first such target position as the current targetposition for which to acquire one or more images, and then continues toperform blocks 442-456 to acquire the images for the target positions atthe current viewing location, including using hardware sensor data todisplay a GUI to guide the image acquisition process.

In particular, in block 442, the routine acquires hardware sensor datafrom the mobile device, such as from one or more IMU units, anddetermines a current position of the mobile device, including a currentorientation and optionally location. In block 444, the routine thenacquires a current image from the mobile device, camera or other imagingsensors. In block 446, the routine then displays the current image onthe display of the mobile device (or other device in use by the user),and overlays the displayed image with a displayed GUI that includes adevice current position visual indicator in the center of the image,target position visual indicators of each of one or more next targetpositions at corresponding locations in the image, and optionallyadditional instructions and/or visual indications related to the imageacquisition process. The routine then continues to block 448, where itdetermines if the current device position matches the current targetposition for which an image is to be acquired, and if so continues toblock 452 to save the current image (or to indicate a current frame ifvideo is being continuously recorded as a user turns at the viewinglocation). If it is instead determined at block 448 that the currentdevice position does not match the current target position, the routinecontinues instead to block 450, where it optionally displays correctiveinstructions in the GUI for the user (e.g., if the current position ofthe mobile device differs from the current target position in an amountor type that exceeds one or more thresholds). After block 452, theroutine continues to block 454, where it determines if there are moretarget positions for which images are to be acquired for the currentviewing location (e.g., based on target positions determined in block440, based on whether a full 360° turn of the user at the viewinglocation has occurred, etc.), and if so continues to block 456 to selectthe next such target position, optionally determining additional targetpositions if all target positions were not determined in block 440.After blocks 450 or 456, the routine returns to block 442 to repeat theprocess and continue to acquire images until all target positions havebeen sufficiently captured.

If it is instead determined at block 454 that images have been acquiredat all of the target positions for the current viewing location, theroutine continues to block 458, where it generates a panorama image forthe viewing location from the saved images or frames, although in otherembodiments the panorama images may be later generated after imageinformation is acquired for all viewing locations in a building or otherstructure. In general, the timing of generating and/or linkingparticular panorama images may be implemented in any order for suchprocessing may be implemented in accordance with the techniquesdescribed herein. For example, the routine may instead processindividual segments of captured information sequentially, such that apanorama image is generated for a first viewing location, followed byprocessing of linking information captured during movement away fromthat first viewing location to determine relative positional informationfor a second viewing location; a panorama image generated for the secondviewing location, followed by processing of linking information capturedduring movement away from that second viewing location to determinerelative positional information for a third viewing location; etc. Invarious embodiments, processing of captured information for one or manybuilding interiors may be performed in a parallel and/or distributedmanner, such as by utilizing one or more parallel processing computingclusters (e.g., directly by the ICA system or via one or morethird-party cloud computing services). After block 458, the routinecontinues to block 460 where it optionally obtains annotation or otherinformation from the user regarding the viewing location, such as toassociate with the generated panorama image for later display to endusers in association with the generated panorama image. After block 460,the routine continues to block 462 and returns, such as to proceed toblock 420 of FIG. 4A.

Returning to FIG. 4A, and if it is instead determined in block 420 thatthere are not any more viewing locations at which to acquire imageinformation for the current building or other structure, the routineproceeds to block 425 to optionally analyze the viewing locationinformation for the building or other structure, such as to identifypossible additional coverage (and/or other information) to acquirewithin the building interior. For example, the ICA system may provideone or more notifications to the user regarding the information acquiredduring capture of the multiple viewing locations and correspondinglinking information, such as if it determines that one or more segmentsof the recorded information are of insufficient or undesirable qualityto serve as the basis for generating a panorama image, or do not appearto provide complete coverage of the building, or would provideinformation for additional inter-panorama links. After block 425, theroutine continues to block 427 to store the generated panorama imagesand any associated generated or obtained information for them, and thento block 430 in order to perform a panorama connection subroutine (withone example of such a routine illustrated in FIGS. 4C and 4D, asdiscussed further below) in order to generate links between some or allof the panorama images generated for the viewing location.

As discussed further in FIGS. 4C and 4D with respect to performing thepanorama connection subroutine for the viewing locations of the buildingor other structure, the routine begins at block 605, where a next pairof panorama images is selected to be analyzed for inter-connectioninformation, beginning with a first pair that includes the first andsecond panorama images corresponding to the first and second viewinglocations in a sequence of multiple viewing locations within a house,building or other structure. The routine then continues to block 610 todetermine whether to attempt to determine connection information betweenthe pair of panorama images via image/feature matching, such as based onoverlap of features in images/frames from the two panorama images, andif so, continues to block 615. It will be appreciated that in someembodiments, connection determination via image/feature matching may notbe performed, such as if all connection information between pairs ofpanorama images is determined using captured linking information, asdiscussed in greater detail with respect to blocks 655-670.

In the illustrated embodiment, the routine in block 615 begins byoptionally filtering pairs of frames/images from the panorama images(e.g., corresponding to individual frames from a video used to constructthe panorama images) that do not have sufficient overlapping coverage,although in other embodiments each image/frame in one of the twopanoramas may be compared to each image/frame in the other of the twopanorama images to determine an amount of overlap, if any, between thepair of images. In the illustrated embodiment, the routine continues toblock 620 from block 615, where it matches non-filtered pairs offrames/images from the two panorama images with overlapping coverage,such as using one or both of essential matrix and/or homography matrixdecomposition processing techniques, although other processingtechniques may be used in other embodiments. In addition, the routinemay optionally select in block 620 whether to retain and use results foreach pair from only one of essential matrix processing and homographymatrix decomposition processing if both are performed, such as dependingon whether information in the pair of frames corresponds to a flatplanar surface or instead as information in a 3D space. In otherembodiments, results from both essential matrix processing andhomography matrix decomposition processing may be retained and used, orinstead only one of the two (and possibly other) types of processing maybe used. The routine further continues in block 620 to determinerelative rotation and translation/distance between the viewing locationsfor the two panorama images from the results of the one or moreprocessing techniques, optionally by combining results from multiplematching image/frame pairs to determine aggregate consensusinter-panorama connection information, and optionally computing aconfidence value in the resulting information.

After block 620, the routine continues to block 625 to determine whetherto attempt to also connect the two panorama images via analysis ofcaptured linking information along a travel path that the user tookbetween the viewing locations corresponding to the two panorama images.If so, or if it is instead determined in block 610 to not attempt toconnect the two panorama images via image matching, the routinecontinues to perform blocks 650-670 to use such linking information todetermine relative rotation and location/direction/distance between thepanorama images. In particular, the routine determines in block 650whether the two panorama images are consecutive images in the sequence,such that linking information is available for a travel path that theuser travels between the two viewing locations corresponding to the twopanorama images, and if not continues to block 630. Otherwise, theroutine continues to block 655 to obtain that linking information forthat travel path, including acceleration data from the mobile device IMUsensor unit(s), and optionally video information as well if available.

After block 655, the routine continues to block 660 to determine thedeparture direction of leaving the viewing location corresponding to thestart panorama image and the arrival direction of arriving at theviewing location of the end panorama image, using video information ifavailable to match initial video information for the departure to one ormore corresponding frames of the start panorama image and to match finalvideo information for the arrival to one or more correspondingopposite-side frames of the end panorama image. If video information isnot available, leaving and arrival directions may be determined in othermanners, such as based solely on analysis of the captured accelerationdata and/or other location information for the mobile device. Afterblock 660, the routine continues to block 665 to analyze theacceleration data in the captured linking information along the travelpath—in particular, for each acceleration data point, a doubleintegration operation is performed to determine first velocity and thenlocation corresponding to that acceleration data point, including in theillustrated embodiment to determine corresponding velocity and locationfor each of x, y, and z axes in three dimensions. In block 670, theroutine then combines the determined velocity and location for each ofthe acceleration data points to form a modeled travel path, along withthe determined leaving/arriving directions, and uses the resultinginformation to determine relative rotation and location/distance betweenthe panorama images, optionally with a corresponding confidence value.

After block 670, or if it instead determined in block 650 that the twopanorama images do not have captured linking information for a travelpath between them, the routine continues to block 630 to, if connectioninformation is available from both image matching and linkinginformation, combine the information into a final determined aggregaterelative direction and distance/location for the panorama images, alongwith the resulting confidence value from the combination. After block630, or if it is instead determined in block 625 to not use linkinginformation to connect the two panorama images, the routine continues toblock 635 to, for each panorama in the pair and based on the determinedrelative position information, determine a direction of the otherpanorama relative to the current panorama starting point, identify oneor more frames in the current panorama that correspond to thatdetermined direction, and store information for the current panoramaabout an inter-panorama link to the other panorama for those one or moreframes.

After block 635, the routine continues to block 645 to determine whetherthere are more pairs of panorama images to analyze, and if so, returnsto block 605 to select the next such pair. In some embodiments, eachconsecutive pair of panorama images in the sequence of viewing locationsis analyzed, and then some or all other pairs of panorama images that donot have corresponding linking information based on a travel pathbetween those viewing locations are considered, so as to determine andprovide inter-panorama connection information for all pairs of panoramaimages for which information is available. As discussed in greaterdetail elsewhere herein, in some embodiments, some links between pairsof panoramas may not be provided even if they may be calculated,however, such as to provide inter-panorama links upon display to an enduser only for a subset of panorama pairs (e.g., corresponding topanorama pairs that are visible to each other, or near each other withina defined distance, or otherwise satisfy one or more specifiedcriteria).

If it is instead determined in block 645 that there are no more pairs ofpanorama images to consider, the routine continues to block 690 tooptionally perform a global review of the respective panorama locationsand the relative directions between them for overall consistency, and toupdate that determined information as appropriate, as discussed ingreater detail elsewhere. If so, such an update may include updating thestored information for one or more panoramas about one or moreinter-panorama links from that panorama to one or more other panoramas.After block 690, the routine continues to block 695 to provideinformation about the determined linked panorama images, and continuesto block 699 and returns to block 435 of FIG. 4A, including to providethe information about the determined linked panorama images.

Returning to FIG. 4A, and after block 430, the routine proceeds to block435 to store the generated information for the building or otherstructure, including to optionally create and store one or moreadditional representations of the building interior, such as by invokingone or more corresponding subroutine. FIGS. 5A-5B illustrate one exampleof a routine for generating a floor map representation of such abuilding interior from the generated and linked panorama information.

If it is instead determined in block 410 that the instructions or otherinformation recited in block 405 are not to acquire images and otherdata representing a building interior, the routine continues instead toblock 490 to perform any other indicated operations as appropriate, suchas any housekeeping tasks, to configure parameters to be used in variousoperations of the system (e.g., based at least in part on informationspecified by a user of the system, such as a user of a mobile device whocaptures one or more building interiors, a user representing an operatorof the system, etc.), to obtain and store other information about usersof the system, to respond to requests for generated and storedinformation, etc.

Following blocks 435 or 490, the routine proceeds to block 495 todetermine whether to continue, such as until an explicit indication toterminate is received. If it is determined to continue, the routinereturns to block 405 to await additional instructions or information,and if not proceeds to step 499 and ends.

FIGS. 5A-5B illustrate an example embodiment of a flow diagram for aMapping Information Generation Manager (MIGM) System routine 500. Theroutine may be performed by, for example, execution of the MIGM system140 of FIG. 1A, the MIGM system 389 of FIG. 3 , and/or an MIGM system asdescribed elsewhere herein, such as to generate and use mappinginformation for a defined area based at least in part on interconnectedimages of the area. In the example of FIGS. 5A-5B, the generated mappinginformation includes a floor map of a building, such as a house, but inother embodiments, other types of mapping information may be determinedand generated for other types of buildings and used in other manners, asdiscussed elsewhere herein.

The illustrated embodiment of the routine begins at block 505, whereinformation or instructions are received. The routine continues to block510 to determine whether the instructions received in block 505 indicateto generate a floor map for an indicated building, optionally along withassociated information about the building, and if so the routinecontinues to perform blocks 530-588 to do so, and otherwise continues toblock 590.

In block 530, the routine obtains existing panoramas or other imagesfrom multiple viewing locations in multiple rooms of the building, alongwith interconnection information for the images and acquisition ofmetadata information related to movement between the viewing locations,such as may optionally be supplied in block 505 along with thecorresponding instructions. After block 530, the routine continues toblock 535 to optionally obtain additional information about thebuilding, such as from one or more external sources (e.g., onlinedatabases, information provided by one or more users, etc.)—suchadditional information may include, for example, exterior dimensionsand/or shape of the building, additional images and/or annotationinformation acquired corresponding to particular locations within thebuilding (optionally for locations different from viewing locations ofthe acquired panorama or other images), etc.

After block 535, the routine continues to block 540 to use the obtainedor acquired image and inner-connection information to determine, for theviewing locations of images inside the building, relative globalpositions of the viewing locations in a common coordinate system orother common frame of reference, such as to determine directions andoptionally distances between the respective viewing locations. Afterblock 540, the routine in block 550 analyzes the acquired or obtainedpanoramas or other images to determine, for each room in the buildingthat has one or more viewing locations, a position within the room ofthose viewing locations. In block 555, the routine further analyzes theimages and/or the acquisition metadata for them to determine, for eachroom in the building, any connecting passages in or out of the room. Inblock 560, the routine then receives or determines estimated room shapeinformation and optionally room type information for some or all roomsin the building, such as based on analysis of images, informationsupplied by one or more users, etc. It will be appreciated that, whileblocks 550-560, are illustrated in separate operations, in someembodiments a single analysis of the images may be performed to acquireor determine multiple types of information, such as those discussed withrespect to blocks 550-560.

In block 565, the routine then separately positions each room shape foreach room around the viewing locations of any images in the room usingthe previously determined relative global position information for theviewing locations, in order to determine initial estimated positions ofthe room shapes. In block 570, the routine then generates finalpositions of each room to create a resulting floor map, includingmatching connecting passages between rooms and optionally applying otherconstraints from one or more of the obtained additional buildinginformation, room shapes and/or room types, other information from theanalysis of the images and/or their acquisition metadata, etc. Such afloor map may include, for example, relative position and shapeinformation for the various rooms without providing any actual dimensioninformation for the individual rooms or building as a whole, and mayfurther include multiple linked or associated sub-maps (e.g., to reflectdifferent stories, levels, sections, etc.) of the building.

After block 570, the routine optionally performs one or more steps 575through 585 to determine and associate additional information with thefloor map. In block 575, the routine optionally estimates the dimensionsof some or all of the rooms, such as from analysis of images and/ortheir acquisition metadata or from overall dimension informationobtained for the exterior of the building, and associates the estimateddimensions with the floor map—it will be appreciated that ifsufficiently detailed dimension information were available, a floor planmay be generated from the floor map. After block 575, the routinecontinues to block 580 to optionally associate further information withthe floor map, such as additional images and/or annotation information,and with particular rooms or other locations within the building. Inblock 585, the routine further optionally estimates heights of some orall rooms, such as from analysis of images and optionally sizes of knownobjects in the images, as well as height information about a camera whenthe images were acquired, and further uses such information to generatea 3D model of the building, with the 3D model further associated withthe floor map.

After block 585, the routine continues to block 588 to store and/orotherwise use the generated floor map information and optionally othergenerated information, such as to provide the generated information fordisplay on one or more client devices, provide that generatedinformation to one or more other devices for use in automatingnavigation of those devices and/or associated vehicles or otherentities, etc.

If it is instead determined in block 510 that the information orinstructions received in block 505 are not to generate a floor map foran indicated building, the routine continues instead to block 590 toperform one or more other indicated operations as appropriate. Suchother operations may include, for example, receiving and responding torequests for previously generated floor maps and/or other generatedinformation (e.g., requests for such information for display on one ormore client devices and/or to provide to one or more other devices foruse in automated navigation), obtaining and storing information aboutbuildings for use in later floor map generation operations (e.g.,information about exterior images, dimensions, numbers or types ofrooms, total square footage, etc.), etc.

After blocks 588 or 590, the routine continues to block 595 to determinewhether to continue, such as until an explicit indication to terminateis received. If it is determined to continue, the routine returns toblock 505 to wait for and receive additional instructions orinformation, and otherwise continues to block 599 and ends.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of thepresent disclosure. It will be appreciated that each block of theflowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer readable program instructions. It will befurther appreciated that in some implementations the functionalityprovided by the routines discussed above may be provided in alternativeways, such as being split among more routines or consolidated into fewerroutines. Similarly, in some implementations illustrated routines mayprovide more or less functionality than is described, such as when otherillustrated routines instead lack or include such functionalityrespectively, or when the amount of functionality that is provided isaltered. In addition, while various operations may be illustrated asbeing performed in a particular manner (e.g., in serial or in parallel,or synchronous or asynchronous) and/or in a particular order, in otherimplementations the operations may be performed in other orders and inother manners. Any data structures discussed above may also bestructured in different manners, such as by having a single datastructure split into multiple data structures and/or by having multipledata structures consolidated into a single data structure. Similarly, insome implementations illustrated data structures may store more or lessinformation than is described, such as when other illustrated datastructures instead lack or include such information respectively, orwhen the amount or types of information that is stored is altered.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the invention. Accordingly, the invention is not limited exceptas by corresponding claims and the elements recited by those claims. Inaddition, while certain aspects of the invention may be presented incertain claim forms at certain times, the inventors contemplate thevarious aspects of the invention in any available claim form. Forexample, while only some aspects of the invention may be recited asbeing embodied in a computer-readable medium at particular times, otheraspects may likewise be so embodied.

What is claimed is:
 1. A computer-implemented method comprising:generating, by a mobile device at a target location in a room of abuilding, a panorama image for the target location from a plurality ofimages acquired in a plurality of target horizontal directions from thetarget location, including: acquiring, by a camera of the mobile deviceas the camera is rotated horizontally at the target location, a firstgroup of the plurality of images for a first subset of the plurality oftarget horizontal directions, including displaying changing first viewsfrom the camera as a current direction of the camera changes at thetarget location, and including displaying to-be-acquired targetdirection visual indicators overlaid on the displayed first views at thetarget horizontal directions of the first subset that identify where toacquire the images of the first group; determining, for each of theacquired images of the first group, a status of that acquired image,wherein one or more of the acquired images of the first group aredetermined to have a successful acquisition status, and wherein one ormore other of the acquired images of the first group are determined tohave an unsuccessful acquisition status; and acquiring, by the camera asthe camera is further rotated horizontally at the target location, asecond group of the plurality of images for a second subset of theplurality of target horizontal directions, including displaying changingsecond views from the camera as the current direction of the camerachanges at the target location and displaying additional targetdirection visual indicators overlaid on the displayed second views,including displaying to-be-acquired target direction visual indicatorsat the target horizontal directions of the second subset that identifywhere to acquire the images of the second group and that visuallyindicate a status of not yet acquired, and displayingpreviously-acquired target direction visual indicators for at least someof the acquired images of the first group that identify where thoseimages of the first group were acquired and that visually indicate thedetermined status of those images of the first group, wherein thedetermined status that is visually indicated for the at least someacquired images of the first group includes the successful acquisitionstatus for at least one of the acquired images and includes theunsuccessful acquisition status for at least one other of the acquiredimages; and displaying at least some of the generated panorama image. 2.The computer-implemented method of claim 1 wherein the determining ofthe status of each of the acquired images of the first group furtherincludes determining whether that acquired image satisfies one or moredefined criteria that are related to quality of contents of thatacquired image and that are based on at least one of an amount oflighting or a speed of rotation of the camera or one or more objectsobscuring a view of the room from the camera.
 3. Thecomputer-implemented method of claim 1 wherein the determining of thestatus of each of the acquired images of the first group is furtherbased on at least one of a recency of acquiring that acquired image or atype of information acquired with that acquired image.
 4. Thecomputer-implemented method of claim 1 wherein the determined status ofeach of the at least some acquired images of the first group is selectedfrom multiple defined statuses, and wherein the displaying of thepreviously-acquired target direction visual indicators for the at leastsome acquired images of the first group includes displaying at least oneof colors from multiple colors each associated with one of the multipledefined status types or patterns from multiple patterns each associatedwith one of the multiple defined status types.
 5. Thecomputer-implemented method of claim 1 wherein the determined status ofeach of the at least some acquired images of the first group is selectedfrom multiple defined statuses, and wherein the displaying of thepreviously-acquired target direction visual indicators for the at leastsome acquired images of the first group includes using at least one ofshapes from multiple shapes each associated with one of the multipledefined statuses or sizes from multiple sizes each associated with oneof the multiple defined statuses.
 6. The computer-implemented method ofclaim 1 wherein the acquiring of the images of the first and secondgroups includes using multiple hardware sensors on the mobile devicethat are part of one or more inertial measurement units, and wherein thegenerating of the panorama image includes generating a panorama imagethat shows an interior of the room from the target location and that has360 degrees of horizontal coverage around a vertical axis.
 7. Anon-transitory computer-readable medium having stored contents thatcause one or more computing devices to perform automated operationsincluding at least: generating, by the one or more computing devices, apanorama image for a target location from a plurality of images acquiredin a plurality of horizontal directions from the target location,including: determining, after one or more first images of the pluralityof images are acquired in one or more first horizontal directions of theplurality of horizontal directions, a status for each of the firstimages that is based at least in part on whether that first image wassuccessfully acquired, wherein the determining of the status for one ofthe first images includes determining that acquisition of the one firstimage was unsuccessful; and acquiring, by a camera at the targetlocation, one or more additional second images of the plurality ofimages for one or more additional second horizontal directions of theplurality of horizontal directions, including displaying changing viewsfrom the camera as a current direction of the camera changes at thetarget location, and including displaying target direction visualindicators overlaid on the displayed views for the first horizontaldirections and the second horizontal directions, wherein the displayedtarget direction visual indicators for the second horizontal directionsidentify one or more second positions at which to acquire the secondimages and include visual information to identify that the second imageshave not been successfully acquired, and wherein the displayed targetdirection visual indicators for the first horizontal directions identifyone or more first positions at which the first images were acquired andinclude visual information to identify the respective determined statusfor each of the first images, including displaying a target directionvisual indicator for the one first image with visual information toindicate that the acquisition of the one first image was unsuccessful;and storing, by the one or more computing devices, the generatedpanorama image for use in sub sequent display.
 8. The non-transitorycomputer-readable medium of claim 7 wherein the stored contents includesoftware instructions that, when executed, cause the one or morecomputing devices to perform further automated operations that includeproviding instructions to initiate a re-acquisition of the one firstimage.
 9. The non-transitory computer-readable medium of claim 8 whereinthe further automated operations include, after a successfulre-acquisition of the one first image, updating the displayed targetdirection visual indicator for the one first image with changed visualinformation to indicate that the one first image was successfullyacquired.
 10. The non-transitory computer-readable medium of claim 7wherein the stored contents include software instructions that, whenexecuted, cause the one or more computing devices to perform furtherautomated operations, including, after acquiring one of the secondimages at one of the identified one or more second positions, updatingthe displayed target direction visual indicator for the one second imagewith changed visual information that reflects the acquired one secondimage.
 11. The non-transitory computer-readable medium of claim 10wherein the updating of the displayed target direction visual indicatorfor the one second image with changed visual information includes usingchanged visual information for that displayed target direction visualindicator that shows that the one second image was successfullyacquired.
 12. The non-transitory computer-readable medium of claim 7wherein the displaying of the target direction visual indicators for thesecond horizontal directions includes choosing at least one of a firstcolor or a first pattern for each of those displayed target directionvisual indicators that represents that the second images have not beensuccessfully acquired, wherein the displaying of the target directionvisual indicators for the first horizontal directions includes choosing,for each of the one or more first images, at least one of a second coloror a second pattern for the displayed target direction visual indicatorfor that first image that represents the respective determined statusfor that first image, and wherein the at least one of the first color orthe first pattern is different from the at least one of the second coloror the second pattern.
 13. The non-transitory computer-readable mediumof claim 7 wherein the displaying of the target direction visualindicators for the first horizontal directions includes choosing, foreach of the one or more first images, at least one of a size or a shapeor a type of element for the displayed target direction visual indicatorfor that first image that represents the respective determined statusfor that first image.
 14. The non-transitory computer-readable medium ofclaim 7 wherein the determining of the status for each of the firstimages includes determining that the one first images was notsuccessfully acquired due to problems with at least one of a location ofthe camera at acquisition of the one first image or an orientation ofthe camera at the acquisition of the one first image.
 15. Thenon-transitory computer-readable medium of claim 7 wherein thedetermining of the status for each of the first images includesdetermining that the one first images was not successfully acquired dueto problems with at least one of insufficient light for an area to becaptured by the one first image or a rotation of the camera that exceedsa defined threshold while acquiring the one first image or an objectthat obscures a view from the camera of at least some of the area to becaptured by the one first image.
 16. The non-transitorycomputer-readable medium of claim 7 wherein the determining of thestatus for each of the first images includes determining that the onefirst images was not successfully acquired due to the one first imagenot satisfying one or more specified criteria.
 17. The non-transitorycomputer-readable medium of claim 16 wherein the determining of thestatus for each of the first images includes determining that another ofthe first images was successfully acquired due to the another firstimage satisfying the one or more specified criteria, and wherein thedisplaying of the target direction visual indicators for the firsthorizontal directions includes using different visual information forthe target direction visual indicator associated with the one firstimage and the target direction visual indicator associated with theanother first image.
 18. The non-transitory computer-readable medium ofclaim 7 wherein the determining of the status for each of the firstimages includes, for another of the first images, determining the statusof the another first image based on at least one of a quality of theanother first image exceeding a defined threshold or a recency ofacquisition of the another first image or an annotation associated withthe another first image.
 19. The non-transitory computer-readable mediumof claim 7 wherein the determining of the status for each of the firstimages includes, for another of the first images, determining the statusof the another first image based at least in part on one or more typesof information captured with the another first image from an environmentaround the target location, wherein the one or more types of informationinclude at least one of audio data or infrared data or ultraviolet dataor cell phone signal data or radiation data or Wi-Fi signal data ormagnetic field data or GPS data.
 20. The non-transitorycomputer-readable medium of claim 7 wherein the camera is part of amobile device that is one of the one or more computing devices, whereinthe target location is in a room of a building, and wherein thedisplaying of the changing views from the camera includes updating thedisplayed views as the camera is rotated at the target location tocapture an interior of the room from the target location, includingusing multiple hardware sensors that are part of one or more inertialmeasurement units on the one or more computing devices to updatepositions of at least one of the one or more target direction visualindicators overlaid on the displayed views.
 21. The non-transitorycomputer-readable medium of claim 7 wherein the target location is in aroom of a building, wherein the displaying of the changing views fromthe camera and the overlaid target direction visual indicators for thefirst horizontal directors and the second horizontal directions is to auser participating in acquiring of the plurality of images, and whereinthe stored contents include software instructions that, when executed,cause the one or more computing devices to perform further automatedoperations including, after the generating of the panorama image for thetarget location: determining, by the one or more computing devices, anadditional target location in the building at which to generate anadditional panorama image, the additional target location being separatefrom and visible from the target location; and displaying additionalinformation that guides the user to the additional target location,including displaying one or more additional views from the camera as thecamera moves from the target location to the additional target locationand including displaying one or more target location visual indicatorsoverlaid on the displayed one or more additional views to indicate theadditional target location.
 22. The non-transitory computer-readablemedium of claim 7 wherein the generating of the panorama image includesforming the panorama image by combining the plurality of images based atleast in part on overlap of content between images of the plurality. 23.A system comprising: one or more hardware processors of one or morecomputing devices; and one or more memories with stored instructionsthat, when executed by at least one of the one or more hardwareprocessors, cause at least one of the one or more computing devices toperform automated operations including at least: generating, for atarget location associated with a building, a panorama image from aplurality of images acquired in a plurality of directions from thetarget location, including: determining, after one or more first imagesof the plurality of images are acquired in one or more first directionsof the plurality of directions, and for each of the first images, astatus of that first image based at least in part on whether that firstimage satisfies one or more defined criteria, wherein the determining ofthe status for one of the first images includes determining thatacquisition of the one first image was unsuccessful; and acquiring, by acamera at the target location, one or more additional second images ofthe plurality of images for one or more additional second directions ofthe plurality of directions, including displaying changing views fromthe camera as a current direction of the camera changes at the targetlocation, and including displaying information overlaid on the displayedviews for the first directions that identify one or more first positionsat which the first images were acquired and that include visualinformation to identify the respective determined status for each of thefirst images, wherein the displaying of the information overlaid on thedisplayed views includes displaying a target direction visual indicatorfor the one first image with visual information to indicate that theacquisition of the one first image was unsuccessful; and storing thegenerated panorama image for subsequent use.
 24. The system of claim 23wherein the determining of the status for another of the first imagesincludes determining the status for the another first image based on atleast one of a quality of the another first image or a recency ofacquisition of the another first image or an annotation associated withthe another first image.
 25. The system of claim 23 wherein thedetermining of the status for another of the first images includesdetermining the status for the another first image based at least inpart on one or more types of data captured with the another first imagefrom an environment around the target location, wherein the one or moretypes of data include at least one of audio data or infrared data orultraviolet data or cell phone signal data or radiation data or Wi-Fisignal data or magnetic field data or GPS data.
 26. The system of claim23 wherein the determining of the status for another of the first imagesincludes determining the status for the another first image based onmetadata associated with the another first image.
 27. The system ofclaim 23 wherein the displaying of the information overlaid on thedisplayed views includes displaying first target direction visualindicators overlaid on the displayed views for the first directions, andfurther includes displaying second target direction visual indicatorsoverlaid on the displayed views for the second directions that identifyone or more second positions at which to acquire the second images andinclude visual information to identify that the second images have notbeen acquired, and wherein the determined status for each of the one ormore first images is based at least in part on whether that first imagehas been successfully acquired.
 28. The system of claim 23 wherein theone or more computing devices include a mobile computing device thatcontains the camera, wherein acquiring of the additional second imagesincludes using multiple hardware sensors on the mobile computing devicethat are part of one or more inertial measurement units, wherein thetarget location is in one of multiple rooms of the building, and whereinthe generating of the panorama image includes generating a panoramaimage with 360 degrees of coverage around a vertical axis to show aninterior of at least part of at least one room of the multiple rooms.29. The system of claim 23 wherein the displaying of the changing viewsfrom the camera and the information overlaid on the displayed views isto a user participating in acquiring of the plurality of images, andwherein the stored instructions include software instructions that, whenexecuted, cause the one or more computing devices to perform furtherautomated operations including: determining an additional targetlocation in the building at which to generate an additional panoramaimage, the additional target location being separate from and visiblefrom the target location; and displaying additional information thatguides the user to the additional target location, including displayingone or more additional views from the camera as the camera moves to theadditional target location and including displaying one or more targetlocation visual indicators overlaid on the displayed one or moreadditional views to indicate the additional target location.